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Abstract 

The notion of entropy penetrates much of science. A key feature 
K^ , of the all-important notion of Boltzmann-Gihhs- Shannon entropy is its 

[~^ ' extensivity (additivity over independent subsystems). However, there 

cn . is a need for other quantities. In statistical physics a parameterized 

family of non-extensive entropy measures, now mainly known under 
^_^ the name of Tsallis-entropies, have received much attention but also 

f— ^ ■ been met with criticism due mainly to a lack of convincing interpre- 

CO , tations. 

Based on the hypothesis that interaction between truth, held by 
"Nature" , and belief, as expressed by man, may take place, classical- as 
K^ , well as non-classical measures of entropy and other essential quantities 

H ' are derived. The approach aims at providing a genuine interpretation, 

rather than relying either on analogies based on formal mathemati- 
cal manipulations or else - more fruitfully, but not satisfactory - on 
axiomatic characterizations. 

1 Contemplation 

Let us apply a philosophical approach and put ourselves in the shoes of 
the physicist, planning to set-up experiments and to engage in associated 
observations. He might argue as follows: 






o 



1: As an expression of my beliefs concerning phenomena I plan to ob- 
serve, I shall assign numbers in [0, 1], typically denoted by the letter y, to 
events associated with the phenomena. Certain, to me unknown numbers, 
likewise in [0,1], and likewise associated with events, express the essence of 
the phenomena, and do not depend on my interference. They are referred to 
as truth- assignments and are, typically, denoted by the letter x. 

2: Any event I may observe entails a certain effort on my part. This 
effort I shall also refer to as individual complexity - "individual" , because it 
is associated with each individual event I could encounter. Before setting up 
experiments, I should determine the effort I am willing to or have to devote 
to any event I may be faced with. This should depend only on the assigned 
belief- value y, and is denoted K,{y). The function k, defined on [0, 1] and with 
values in [0,oo], I refer to as the coder. As 1 represents certainty, I insist 
that /t(l) = 0. Further, I assume that k is smooth in a technical sense, say 
continuous on [0, 1] and continuously different iable and finite valued on ]0, 1]. 
Finally, as I do not want to distinguish between coders that only differ by a 
scalar factor, I will introduce an assumption of normalization in order to pick 
out a canonical representative of the possible coders. As k{1) = and as I 
do not want to assume that /t(0) is finite, I choose to impose the condition 
/t'(l) = —1 for the stated purpose. 

3 To determine the coder, I must know the basic characteristica of the 
world I operate in. I choose to focus primarily on a concept of interaction 
between truth and belief. 

4 I shall model this interaction by a function n defined on the product 
set [0,1] X [0,1] and taking values in ] — oo,oo]. The idea is that 7r{x,y) 
represents the force by which the world presents an event to me in case the 
truth-assignment is x and my belief in the event is y. On the technical 
side, I better assume that n is continuous on its domain and continuously 
differentiable and finite- valued on [0, l]x]0, 1]. 

5 I consider the classical world to be a world of "no interaction", i.e. 
7r(x,2/) = X for all {x,y). I must be prepared for other forms of interaction, 
but will always assume that the interaction is sound, i.e. that 7r(x, x) = x 
for all X G [0, 1]. Stronger conditions should be considered and in this con- 
nection, it appears sensible to impose conditions of consistency: 1 will call 
the interaction weakly consistent if X]jgA'^(-^i'^«) ~ -'- ^'-'^ ^^^ finite alphabet 
A and any truth-assignment x = {xi)i(zj^ and belief-assignment y = {yi)i<z^. 



both assumed to be probability distributions over A. If, with the same con- 
ditions on X and y, it can be concluded that (7rj)jgA = (7i"(a;j, |/j))iGA is in 
fact a probability distribution, just as x and y, 1 will say that n is strongly 
consistent. 

6 To enable observations from the world, I must configure all available 
resources such as observation- and measuring devices. The resulting con- 
figuration will enable me to perform experiments, i.e. to study individual 
situations from the world which have my interest. 

7 Before actual observations are performed, I must identify the various 
possible basic events (or pure states, as some may prefer), which I could 
encounter. I shall characterize them by an index, typically i, intended to 
have semantic significance. The set of possible basic events is the alphabet 
pertaining to the situation, call it A. The actual naming of members of A, 
the semiotic assignment, should catalyze semantic awareness and facilitate 
technical handling. 

8 I will apply a principle of separability and consider my total effort related 
to observations from the configured situation to be the sum of individual 
efforts associated with the basic events. In so doing, I must weigh each 
contribution according to the force with which I will experience the associated 
basic event. The total effort I also refer to as total complexity or simply 
complexity and thus find that complexity is the weighted sum of individual 
complexities: 

^{^^ y) = ^ ^(^i' ydf^ivi) ■ (1) 

ieA 

Here, x = (xj)jgA and y = {yi)i^A are, respectively the truth-assignments and 
the belief-assignments associated with the various basic events. 

9 I will attempt to minimize complexity and shall appeal to the principle 
that the smallest value for complexity is obtained when belief matches truth. 

As 

'^7i{xi,y,i)K{y,i) -^XiK{xi) (2) 

ieA ieA 

represents my frustration, the principle says that frustration is the least, in 
fact disappears, when yi = Xi for all i G A. 

Given x = (xi)igA, minimal complexity is what I will aim at. It is an 
important quantity. In anticipation, I will call it entropy and denote it by 



the letter M 



H(x) = inf $(x, y) = 2^3;i/€(xj) . (3) 



The quantity ([2]) too appears important. It is tempting to call it "frustration" 
but, again in anticipation, I better call it divergence. I shall denote it by the 
letter D: 

D(x,2/) = $(x,2/)-H(a;). (4) 

2 Conclusion 

Theorem 1. With assumptions and definitions as introduced above, assum- 
ing only that the interaction is weakly consistent, the number q = n{l, 0) must 
be non-negative and, to each q G [0,oo[, there is only one pair of interaction 
and coder which fulfill the conditions imposed. These functions, denoted iTq 
and Kg, are determined by the formulas 

■^q{x,y) = qx + {l-q)y, (5) 

Hq{y) = Ing - , (6) 

where the q-logarithm is given by 

Note that strong consistency holds if and only if < g < 1. 
The accompanying quantities, complexity, entropy and divergence are 
denoted $g, H^ and Dg, respectively, and given through ([1]), ([3]) and (jlj), i.e. 

^qi^^ y) =^ '^qi^i^ Vdl^qiVi) > (8) 

^qix) =^XiKg{Xi) (9) 

JGA 

Dg(a;, y) = ^ [T^qixi, yi)Kg{yi) - XiKg{xi)J . (10) 



^In order to allow a singular case - the case g = of Theorem 1 below - to fit into the 
framework, the infimum in ([3]) should be restricted to run over probability distributions y 
with a support which contains the support of x. 



In dn]) we recognize the family of Tsallis entropies, cf. Tsallis [B]. 

Regarding the proof of Theorem 1, we shall here only give a brief in- 
dication: The formula ([5]) is readily derived from the assumption of weak 
consistency. Then, the only possible form for the coder, ([6]), is derived from 
pretty standard variational arguments. The final step of the proof, that with 
(jSj) and (jH]) the variational principle does indeed hold, follows by observing 
the close tie to entropy- and divergence- measures as derived by an approach 
due to Bregman, cf. the recent papers [5] and [^ and references there. 

3 Hints to the literature 

Regarding the formula (jH]) for entropy and its significance, we note that it 
first appeared in the mathematical literature in Havrda and Charvat [1], that 
it then appeared in the physical literature in Lindhard and Nielsen [3] , and in 
Lindhard [2], and that it was efficiently promoted in the paper by Tsallis [6\ 
which triggered much research in the physical community as also witnessed 
by the many entries in the database pointed to under [6j. 

4 Formal publication 

The present manuscript, posted on the arXiv server, is an announcement, 
prior to formal publication. A manuscript with a comprehensive discussion, 
with a full proof of Theorem 1 and with some further results will be worked 
out soon and submitted to the electronic journal "Entropy". 

Further discussion of the considerations presented as well as suggestions 
of concrete mechanisms behind the concept of interaction are among obvious 
issues to look closer into. 

References 

[1] J. Havrda and F. Charvat. Quantification method of classification pro- 
cesses. Concept of structural a-entropy. Kybernetika, 3:30-35, 1967. 

[2] J. Lindhard. On the Theory of Measurement and its Consequences in 
Statistical Dynamics. Mat. Fys. Medd. Dan. Vid. Selsk., 39(l):l-39, 1974. 



[3] J. Lindhard and V. Nielsen. Studies in Statistical Dynamics. Mat. Fys. 
Medd. Dan. Vzd. Selst, 38(9):l-42, 1971. 

[4] J. Naudts. Generalised exponential families and associated entropy func- 
tions. larXiv:0803.0T04V l[matli-ph] . 2008. 

[5] F. Tops0e. Exponential Families and MaxEnt Calculations for Entropy 
Measures of Statistical Physics, in volume 965 of AIP Conference Pro- 
ceedings, pages 104-113, 2007. 

[6] C. Tsallis. Possible generalization of Boltzmann-Gibbs statistics. J. Stat. 



Physics, 52:479-487, 1988. See |littp://tsallis.cat.cbpf.br/biblio.litm| for 
an updated bibliography. 



